Leading Indicators DAP

Section 1: Motivation

Vishnu’s section

Section 2: Methodology

Initial exploration: Cross Correlation

Our first approach to this DAP looked at the relationship between the indicator and the signal more generally. We first used cross correlation analysis on the time series to identify the relationship between indicators and cases across a time period. We calculated the cross correlation of the two time series for each county, and looked at the max cross correlation values and the lag in days for where that maximum cross correlation occured. An example of this data over all observed counties for the Drs Visits indicator signal:

However, this is a more general analysis than our DAP directive, which is dealing with periods of increase specifically, rather then the general relationship between two signals. For this DAP, we don’t care if there is a cross correlation or not between indicators and cases when the signals are flat, decreasing, or even rising slightly. We care about sharp upswings in cases, something significant within a county. We want to analyze the leadingness of a signficant indicator rise in relation to a significant case rise. To do this, we need to define and find periods of significant rise in these signals.

Identifying Rises in Signals

We determined that to ascertain leadingness of an indicator, we care about whether the indicator began to rise significantly before cases began to rise significantly. So we need a decision rule for identifying significant rises.

Starting at the Peak
At first we experimented with finding the peak of a signal in a given time period and identifying the closest local minimum that precedes the peak. However this is not always the point at which the signal actually begins to rise (could be caught in a shallow local minimum), and does not gaurantee the rise would be a lengthy or a steep one. This method also only picks one rise period for a signal for every county for the given time period, which isn’t always reflective of the signal’s actual behavior.

Best Fit Line
One option we tried was calculating a line of best fit for the signal for fixed time periods within a larger time period. For example, calculating a line of best fit for every 21 day window within a 3 month window and choosing the period that has the highest slope as the most significant rise period in that county for that signal.

• Where this worked well:
  • This method finds periods of consistent rise, not allowing the signal to vary a lot between the beginning and ending points of the rise period.
• Where this had drawbacks:
  • Many times this method only selects the larger end of a rise, since the window is a fixed size, and leaves out where the rise starts.
  • This method also only allows for one rise per time period/county.

Example plot

Estimated Derivative
We then tried using multiple different derivative estimate methods to identify periods where the estimated derivative at each point is over a certain threshold.

• Where this worked well:
  • It can find smaller periods of continuous rise that are flexible in length.
  • It allows for multiple periods of rise per time period.
• Where this had drawbacks:
  • Using a large fixed window size (14 or 21 days) in the derivative estimation function means that many identified points are actually within a decreasing period after a rise.
  • It can label rises too liberally, identifying very small bumps in the signal as rises.

Example plot

Blue is using smoothing spline method, Red is using local linear regression (Purple is where both methods marked the same point)

Final Method

We saw that smoothing the signal first (on top of the 7 day average smoothing from the API) and using the derivative method produced the best results. Twekaing this method with some other decision rules gave us our best outcome for finding periods of significant rise.

Final criteria for rise periods: A period is a significant rise if

1. First derivative at each point is > 0 - aka signal is in fact rising on every day

2. Period is > a certain number of days (for this analysis we used 5) - aka it’s not just a spurius rise

3. Each first derivative is > a certain % of other derivatives in time period (for this analysis we used 75%) - aka making sure the rise is a significant one for this county. This ties this decision to the specific time period we are looking at, and the rise point identifications can change based on the time period because of this criteria.

4. Magnitude of increase from start to end of period is > a certain threshold (for this analysis we used 20%) - aka another way to make sure that the rise is significant, not just a slight uptick in cases

Finally, we take the point at the beginning of each rise period as the best estimation of a point of inflection where a signal begins to rise significantly, so we can answer the question: Does the beginning of a rise in the indicator come before the beginning of a rise in cases?

In our analysis, we look at this on a county by county basis, for specific time periods during the pandemic, like the “Summer wave” or the “Fall wave.”

County Selection

In our analysis, we include all counties that have greater than 2000 cases (a little over 20 cases a day for a 3 month window), 80 days of indicator data for a 3 month window, and do not have zero or negative values for either cases or the indicator.

Recall and Precision

TODO

Walkthrough

Step 1. Get and prepare county data

As an example, we’ll use our Dr Visits % CLI as our indicator, and the summer as our time period. We use our LeadingIndicatorTools package for all our main functions.

Step 2. We can plot the Drs Visits and the case signal together for an example county.

plot_signals(drs_summer, "01003", smooth_and_show_increase_point=FALSE, "Drs Visits")

Step 3. Now we can mark the rise points (the points at the beginning of the rise periods) for the Drs Visits and Cases signal.

In the respective rise point columns, the day is marked with a 1 if it is found to be a rise point for that signal. We can see here that there is a rise point for Drs Vists on 6/18 and for cases on 6/26.

drs_summer[1]

[[1]]

NA

Step 4. Now we can plot the smoothed signal with the beginning rise points.

We can see that Drs Visits begins to rise before cases rise. TODO I think we need to tweak our rise point method a bit so we don’t have these “double counting” points on a rise.

plot_signals(drs_summer, "01003", smooth_and_show_increase_point=TRUE, "Drs Visits")

Section 3: Analysis

Doctor Visits

We can plot some of the counties where rises in the doctor visits indicator consistently lead rises in cases.

Examples from the Summer

Some inputs were not uniquely matched; returning only the first match in each case.

Examples from the Fall

Some inputs were not uniquely matched; returning only the first match in each case.

Examples of counties that show successes in both Summer and Fall

Some inputs were not uniquely matched; returning only the first match in each case.Some inputs were not uniquely matched; returning only the first match in each case.

Fall Frequencies

We can also look at the distribution of the frequency of the number of days by which Doctor Visits’ rises lead case rises in successful counties

Summer Frequencies

We can also look at the distribution of the frequency of the number of days by which Doctor Visits’ rises lead case rises in successful counties

Change Healthcare

We can plot some of the counties where rises in the Change Healthcare indicator consistently lead rises in cases.

Examples from the Summer

Some inputs were not uniquely matched; returning only the first match in each case.Some inputs were not uniquely matched; returning only the first match in each case.Some inputs were not uniquely matched; returning only the first match in each case.

Examples from the Fall

Examples of counties that show successes in both Summer and Fall

Fall Frequencies

We can also look at the distribution of the frequency of the number of days by which Change Healthcare rises lead case rises in successful counties

Summer Frequencies

We can also look at the distribution of the frequency of the number of days by which Change Healthcare rises lead case rises in successful counties

Indicator Combination

We can plot some of the counties where rises in the indicator combination consistently lead rises in cases.

Examples from the Summer

Some inputs were not uniquely matched; returning only the first match in each case.Some inputs were not uniquely matched; returning only the first match in each case.

Examples from the Fall

Examples of counties that show successes in both Summer and Fall

Fall Frequencies

We can also look at the distribution of the frequency of the number of days by which Indicator Combination rises lead case rises in successful counties

Summer Frequencies

We can also look at the distribution of the frequency of the number of days by which Indicator Combination rises lead case rises in successful counties

Note that the counties shown and counted as “successful” here met the following criteria: All indicator rise points were followed by a case rise point within 3 to 14 days (and all case rise points were preceded by an indicator rise point within the same time period). This means that the displayed counties almost always have only one rise point per signal and case (the more rise points the harder it is to meet the criteria).

Section 4: Performance

Recall and precision.

Section 5: Conclusions and Limitations